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Abstract 


The  Standard  CMMIsm  Appraisal  Method  for  Process  Improvement  (SCAMPIsm)  V  1.1  is 
designed  to  provide  benchmarks  relative  to  Capability  Maturity  Model®  Integration  (CMMI) 
models.  This  appraisal  method  is  applicable  to  a  wide  range  of  appraisal  applications, 
including  support  for  both  internal  process  improvement  and  external  capability 
determination. 

The  SCAMPI  VI.  1  Method  Definition  Document  describes  the  requirements,  activities,  and 
practices  associated  with  each  of  the  processes  that  compose  SCAMPI.  This  technical  note 
provides  additional  implementation  guidance  related  to  supplier  selection  and  contract 
process  monitoring  applications  of  this  appraisal  method.  This  technical  note  does  not  cover 
current  issues  such  as  the  definition  of  “independently  led”  appraisals  or  the  registration  and 
reuse  of  appraisal  results. 


SM  CMMI  is  a  service  mark  of  Carnegie  Mellon  University. 

SM  SCAMPI  is  a  service  mark  of  Carnegie  Mellon  University. 

®  Capability  Maturity  Model  is  registered  in  the  U.S.  Patent  and  Trademark  Office. 
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1  Background 


Guidance  for  the  use  of  appraisal  methods  utilizing  Capability  Maturity  Models  in 
acquisitions  has  its  roots  in  the  Department  of  Defense  Directive  (DoDD)  5000  series  and  in 
Office  of  the  Secretary  of  Defense  (OSD)  policy.  Software  Capability  Evaluations  (SCEssm) 
based  upon  the  Capability  Maturity  Model  for  Software  (SW-CMM®)  and  Software 
Development  Capability  Evaluations  (SDCEs)  have  been  routinely  used  in  supplier  selection 
and  contract  process  monitoring  activities  for  a  number  of  years.  Commercial  industry  began 
using  SCEs  for  analogous  subcontractor  selection  and  monitoring  more  recently  with 
analogous  experience  and  results. 


1 .1  Software  Capability  Evaluations 

Traditionally,  capability  evaluations  have  helped  acquisition  managers  achieve  the  following 
goals: 

•  Identify  risks  by  evaluating  process  capability  in  supplier  selection. 

•  Manage  risk  by  motivating  contractors  to  improve  their  development  processes  without 
forcing  compliance  to  specific  practices. 

•  Monitor  award  fee  incentives  for  contractors  who  have  structured  process  improvement 
programs. 

The  use  of  “external  capability  evaluations”  as  a  means  of  independent  validation  of 
organizations’  development  process  maturity  and  capability  by  commercial  and  government 
organizations  with  oversight  responsibilities  (e.g..  Government  Accounting  Office  and 
Defense  Management  Contracting  Agency)  has  been  steady  and  routine. 


1.2  SCAMPI 

CMMI  and  its  associated  appraisal  method,  SCAMPI,  are  slated  to  replace  the  SW-CMM  and 
its  associated  appraisal  methodologies:  CMM  Based  Appraisal  for  Internal  Process 
Improvement  (CBA-DPI),  VI. 2  and  Software  Capability  Evaluation  (SCE),  V3.0.  This 
replacement  is  expected  to  be  complete  by  the  end  of  2005.  SCAMPI  is  expected  to  be  the 
single  appraisal  methodology  to  be  appropriately  tailored  for  use  with  CMMI  models. 


SM  SCE  is  a  service  mark  of  Carnegie  Mellon  University. 

®  CMM  is  registered  in  the  U.S.  Patent  and  Trademark  Office. 
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SCAMPI  fulfills  the  Appraisal  Requirements  for  CMMI  (ARC)  V 1. 1  document  that  required 
an  appraisal  methodology  capable  of  benchmarking  process  improvement  efforts  internal  to 
contractor  organizations  as  well  as  to  government  and  commercial  supplier-selection  and 
contract-monitoring  applications.  As  a  benchmarking  appraisal  method,  SCAMPI  is 
classified  as  a  class  A  method.  Class  B  and  C  methods  have  less  stringent  appraisal 
requirements. 

With  the  advent  of  SCAMPI  V  1.1,  the  historical  use  of  the  terms  “assessment”  and 
“evaluation”  are  replaced  by  the  term  “appraisal.”  What  were  formerly  assessments  and 
evaluations  will  instead  use  the  same  core  appraisal  methodology  with  tailoring  guidance 
appropriate  to  the  circumstances  of  internal  process  improvement,  supplier  selection,  or 
contract  process  monitoring.  Although  the  ARC  defines  three  classes  of  appraisals  (e.g..  A,  B, 
and  C),  it  is  beyond  the  scope  of  this  technical  note  to  address  appraisal  classes  other  than  the 
benchmarking  methodology  of  SCAMPI  V  1.1  at  this  time. 

Much  of  the  material  in  this  technical  note  was  derived  from  the  Software  Capability 
Evaluation,  V3.0  Implementation  Guide  for  Supplier  Selection  [Barbour  1995]. 
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2  Overview  of  SCAMPI  for  Supplier  Selection  and 
Contract  Process  Monitoring 


SCAMPI  typically  will  be  used  in  two  different  acquisition  environments:  source  selection 
and  contract  process  monitoring.  Supplier  source  selection,  the  application  for  which  SCE 
was  originally  developed  and  which  SCAMPI  will  replace,  has  been  in  routine  use  since  the 
original  publication  of  the  CMM  concepts.  Current  trends,  however,  have  seen  a  consistent 
application  of  SCEs  in  the  post-contract  award  environment.  Similarly,  the  commercial 
community  has  been  applying  SCEs  in  the  selection  of  subcontractors  and  teaming  partners. 

It  is  expected  that  the  demand  in  these  environments  will  continue,  but  these  demands  will  be 
satisfied  by  the  application  of  SCAMPI  instead  of  SCEs. 

Factors  to  consider  before  using  SCAMPI  in  an  acquisition  include  the  following: 

•  How  critical  is  the  component? 

•  Do  you  lack  data  about  the  offeror’s  past  performance  or  product  development 
capability? 

•  What  is  the  total  dollar  value  of  the  acquisition  or  component? 

•  What  is  the  priority  of  management  control  in  this  acquisition? 

•  Are  the  mission  needs  unprecedented? 

•  What  is  the  current  acquisition  life-cycle  phase? 

•  What  is  the  length  of  time  needed  for  the  acquisition? 

•  What  is  the  size  of  the  acquired  component,  including  the  number  of  configuration  items? 

•  How  good  is  the  relationship  between  the  prime  contractor  and  subcontractor? 

2.1  SCAMPI  in  Supplier  Selection 

The  factors  listed  above  affect  the  implementation  of  SCAMPI  and  become  visible  in  the 
following  acquisition  documentation: 

•  Commerce  Business  Daily  or  similar  announcement 

•  source  selection  plan  (SSP) 

•  evaluation  plan  (EP) 

•  bidder’s  briefing 

•  request  for  proposal  (RFP) 
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•  statement  of  objectives  or  award  fee  plan  (possibly) 

•  briefing  to  successful  offeror 

•  briefing  to  unsuccessful  offerors 

When  used  effectively,  virtually  every  major  activity  in  a  source  selection  is  affected  by 
SCAMPI.  Each  of  these  documents,  particularly  the  SSP,  EP,  and  RFP,  facilitates  the  use  of 
SCAMPI  during  the  source  selection  evaluation.  Figure  1  illustrates  a  global  view  of  a 
representative  source  selection  schedule  that  includes  SCAMPI  activities. 


Phase  I  Phase  II  Phase  III 


Decision  to  use  RFP  is  Proposals 

SCAMPI;  SSP  released  are  received 

evaluation 


The  SSEB  The  SSAC  performs  a  The  SSA  makes  the 

evaluates  the  risk/benefit  assessment  award  decision 

offerors 


For  each  offeror 

•  Determine,  prioritize,  and  finalize  the  data  collection  plan  (1*5  days) 

•  Visit  the  site  (5-10  days) 

•  Produce  a  findings  report  (2-3  days) 


Figure  1:  SCAMPI  Activities  in  a  Typical  Supplier  Selection  Timeline 

The  following  paragraphs  describe  each  phase  of  the  timeline  using  SCAMPI  for  supplier 
selection. 


2.1 .1  Phase  I  -  Decision  Point  to  RFP  Release 

The  decision  to  use  SCAMPI  immediately  sets  things  in  motion  for  appraisal  planning  and 
implementation.  Nominally,  the  decision  is  articulated  in  the  source  selection  plan,  and 
detailed  usage  of  the  determination  of  SCAMPI  evaluation  results  is  delineated  in  the  source 
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selection  evaluation  plan.  Appropriate  language  is  selected  and  tailored  for  insertion  in  the 
request  for  proposal  requiring  SCAMPI  usage  and  how  the  offerors  are  to  provide  SCAMPI- 
related  information  to  the  source-selection  team.  Selecting  the  team  lead  appraiser  and 
SCAMPI  team  members  and  training  them  will  normally  occur  prior  to  proposal  receipt 
because  the  SCAMPI  team  is  not  necessarily  in  place  prior  to  RFP  release.  However,  a 
SCAMPI-knowledgeable  person  is  needed  to  plan  and  prepare  the  RFP  for  the  SCAMPI 
evaluation. 


2.1 .2  Phase  II  -  RFP  Release  to  Proposal  Receipt 

This  phase  of  the  supplier  selection  timeline  is  an  opportunity  to  bring  the  SCAMPI  team 
together  (if  not  already  done),  provide  SCAMPI  team  training,  and  familiarize  the  team  with 
program  requirements  and  risk  areas. 


2.1 .3  Phase  III  -  Proposal  Receipt  to  Site  Visit 

Following  proposal  receipt,  the  evaluation  team  determines  the  specific  data  collection  plan 
to  be  carried  out  for  each  offeror  remaining  in  the  competitive  range  of  the  source  selection. 
An  appraisal  plan  defines  the  organizational  scope  as  well  as  the  CMMI  model  scope  that  are 
the  precursors  for  defining  the  explicit  data  collection  strategy.  Definition  of  these  factors 
includes  specific  determination  of  which  CMMI  representation,  staged  or  continuous,  would 
be  used;  the  targeted  maturity  or  capability  levels;  and  the  process  areas  that  will  be 
evaluated.  During  the  onsite  period,  the  team  collects  information  and  turns  the  information 
into  findings  in  the  form  of  strengths,  weaknesses,  and  improvement  activities  (if  requested 
by  the  sponsor).  At  a  minimum,  ratings  of  process  area  (PA)  goals  based  on  practice 
implementation  characterizations  of  the  organizational  unit  within  the  scope  of  the  appraisal 
will  be  determined  and  reported  as  part  of  the  appraisal  disclosure  statement  (ADS).  (See  the 
SCAMPI  VI .  1  Method  Definition  Document  Appendix  A  for  a  description  of  an  ADS.)  The 
data  and  findings  are  then  provided  to  the  sponsoring  organization  in  the  format  agreed  upon. 

In  most  source  selections,  the  SCAMPI  team  is  one  of  a  number  of  teams  involved  in 
providing  evaluation  services  to  the  Source  Selection  Evaluation  Board  (SSEB).  Typically, 
there  are  other  teams  evaluating  criteria  in  management,  cost,  and  other  technical  areas. 
These  teams  provide  their  findings— just  as  the  SCAMPI  team  provides  their  findings — 
according  to  the  SSP  and  SSEB. 

For  example,  the  SSEB  evaluates  development  organizations’  proposals  for  an  acquisition 
relative  to  a  prescribed/published  set  of  evaluation  criteria  and  identifies  the  risks  (relative  to 
the  evaluation  criteria)  of  development  organizations  being  able  to  fully  execute  a  contract  if 
awarded  to  them.  This  risk  assessment  (relative  to  the  evaluation  criteria)  is  provided  to  the 
Source  Selection  Advisory  Council/Committee  (SSAC). 
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The  SSAC’s  responsibility  is  that  of  overall  risk  assessment  of  suppliers  in  the  competitive 
range.  The  SSAC  assesses  the  overall  risks  of  selecting  each  offeror  and  provides  their 
assessment  of  risk  to  the  Source  Selection  Authority  (SSA),  which  is  empowered  to  make  an 
award  of  an  executable  contract.  The  SSA’s  responsibility  is  to  make  an  award  decision  that 
minimizes  risks  and  maximizes  the  benefits  to  the  sponsor. 


2.2  SCAMPI  in  Contract  Process  Monitoring 

The  value  of  implementing  SCAMPI  in  source  selection  can  continue  past  the  contract  award 
and  into  contract  performance.  The  source-selection  SCAMPI  identifies  a  set  of  risks 
associated  with  the  successful  offeror.  Those  same  risks,  defined  as  weaknesses  associated 
with  individual  process  areas,  can  be  tracked  or  monitored  as  the  contract  progresses  if  the 
program  office  feels  that  improvement  in  those  areas  will  benefit  program  development. 
Monitoring  improvements  can  be  done  by  doing  the  following: 

•  using  weaknesses  to  define  the  risks 

•  developing  a  plan  to  mitigate  the  risks 

•  performing  tradeoff  analysis  to  establish  levels  of  surveillance  for  strong  or  weak  areas 

•  defining  the  adequate  reporting  or  insight  to  be  provided  to  the  program  office  to 

facilitate  monitoring 

In  contemplating  using  SCAMPI  as  a  contract-process-monitoring,  risk-management  tool,  the 
following  questions  could  be  considered: 

•  What  would  you  like  (and  need)  to  know  at  the  start  of  the  contract? 

•  What  expertise  would  the  program  office  need  to  monitor  performance? 

•  What  action  should  be  taken  at  the  start  of  the  contract? 

•  What  action  should  be  taken  if  identified  risks  occur? 

•  Should  there  be  incentives  to  motivate  mitigation  of  the  identified  risks,  possibly  through 
vehicles  such  as  the  program  award  fee? 


Use  SCAMPI  data  to  define  the  risks  associated  with  the  execution  of  the  contract,  to  develop 
a  plan  to  mitigate  those  risks,  and  to  work  the  plan.  This  plan  could  include  such  items  as 
trading  off  the  surveillance  of  strong  areas  for  weak  ones.  If  an  organization  is  found  to  have 
excellent  configuration  management  procedures,  it  is  wasteful  to  check  on  this  process  area 
in  the  same  way  that  would  be  applied  to  an  area  found  to  be  weak  (e.g..  Project  Monitoring 
and  Control). 
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2.3  Using  SCAMPI  to  Baseline  Performance 

As  has  been  done  historically  with  SCEs,  SCAMPI  can  be  used  to  establish  baseline  contract 
process  performance.  One  strategy  that  could  be  used  is  to  establish  a  baseline  of  the 
development  organization’s  performance  relative  to  a  CMMI  model.  The  creation  of  this 
baseline  entails  a  number  of  planning  and  execution  factors. 


2.3.1  For  New  Contracts 

The  salient  points  to  be  integrated  into  a  plan  for  use  of  SCAMPI  for  new  contracts  are: 

•  The  RFP  must  identify  SCAMPI  for  use  in  contract  process  monitoring  (i.e.,  perform  a 
SCAMPI  evaluation  in  source  selection  and  then  use  SCAMPI  to  monitor  the  contract). 

•  SCAMPI  is  still  an  evaluation  factor  in  selection. 

•  Contract  process  monitoring  criteria  should  be  based  on  program  office  needs  and  be 
identified  in  the  contract  or  statement  of  work  (SOW).  The  following  issues  should  be 
assessed  for  inclusion  in  the  contract  or  statement  of  work: 

-  Eliminating  weaknesses 

-  Creating  additional  strengths 

-  Improving  actual  versus  planned  tracking  of  improvement  activities 
within  process  areas 

2.3.2  For  Existing  Contracts 

The  salient  points  to  be  integrated  into  a  plan  for  use  of  SCAMPI  for  existing  contracts  when 
SCAMPI  can  be  used  as  a  contract  process  monitoring  tool  are: 

•  Using  SCAMPI  for  contract  process  monitoring  can  be  a  negotiated  change  to  the 
contract. 

•  When  a  long-term  relationship  is  expected  and  the  benefits  of  process  improvement  can 
be  realized,  SCAMPI  is  a  good  choice  for  contract  process  monitoring. 

•  Refer  to  the  same  criteria  as  for  new  contracts  (above). 

2.3.3  Award  Fees 

Establishing  a  process  baseline  lends  further  utility  of  SCAMPI  in  contract  process 
monitoring  for  considering  award  fees  or  value  engineering  incentives  for  process 
improvement.  Note,  however,  award  fee  applications  (e.g.,  an  award  for  meeting  specified 
measures  of  performance)  are  not  appropriate  in  all  instances.  The  award  fee  application  of 
SCAMPI  is  most  appropriate  under  the  following  circumstances: 

•  A  long-term  relationship  is  involved. 

•  The  contractor  lacks  a  sufficient  number  of  programs  over  which  to  spread  improvement 
costs. 
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•  Process  investments  in  general  would  not  otherwise  be  made. 

•  The  sponsoring  organization  believes  direct  investment  incentives  will  be  the  best 
motivator  of  action. 

•  The  program  environment  includes: 

-  mission-critical  systems 

-  embedded  systems 

-  a  history  of  system/software  engineering  issues 

•  SCAMPI  is  used  by  the  sponsoring  organization  to  mitigate  risks. 

•  The  objective  and  ultimate  goal  of  applying  SCAMPI  are  the  following: 

-  Objective:  Provide  incentive  for  contractors  to  improve  the  total  systems  engineering 
and  software  development  process. 

-  Goal:  Exceed  the  product  development  quality,  cost,  and  schedule  requirements. 

2.3.4  A  Teaming  Approach 

The  sponsoring  organization  and  contractor  should  view  themselves  as  team  members  in  an 
effort  to  benefit  from  an  overall  systems  engineering  and  software  engineering  process 
improvement  plan.  This  teaming  approach  has  some  specific  characteristics: 

•  CMMI  is  the  basis  for  the  improvement  effort. 

The  contractor  uses  CMMI  to  establish  plans. 

The  sponsoring  organization  evaluates  its  processes  using  CMMI. 

•  Contract  incentive  is  the  contractual  vehicle. 

The  contract  describes  the  sponsoring  organization’s  goals. 

The  contract  describes  the  method  of  evaluating  progress. 

•  The  sponsoring  organization  and  contractor  jointly  agree  to  the  criteria  and  approach. 

•  The  award  fee  plan  increments  and  criteria  support  long-range  objectives. 

The  award  fee  plan  increments  and  criteria  can  be  tailored  specifically  to  program 
needs. 

SCAMPI  is  used  to  establish  baseline  systems/software  engineering  process 
capability. 

Findings  are  provided  to  the  contractor. 

The  contractor  uses  findings  to  focus  the  improvement  plan. 

The  sponsoring  organization  and  contractor  jointly  agree  to  goals. 

SCAMPI  is  then  used  to  measure  progress  against  the  improvement  plan. 

Incentive  awards  are  determined  by  the  contract  provisions. 

Findings  establish  the  new  baseline  for  the  next  increment. 


8 


CMU/SEI-2002-TN-008 


The  keys  to  the  successful  application  of  award  fee  usage  of  SCAMPI  are  to  perform  the 
source-selection  SCAMPI  evaluation,  use  the  findings  to  frame  the  award  fee  plan,  perform  a 
baseline  SCAMPI  evaluation  (after  a  suitable  time  frame  [six  months]  that  allows  the 
contractor  to  begin  contract  performance),  have  the  contractor  submit  a  process  improvement 
plan  (PIP),  and  involve  the  contractor  to  obtain  an  understanding  of  the  SCAMPI  findings 
and  impacts  upon  the  award  fee  pool. 


2.3.5  Value  Engineering 

Value  engineering  for  product  development  process  improvement  is  another  mechanism  that 
is  available.  Value  engineering  is  described  in  the  Federal  Acquisition  Regulations  (FAR)  Part 
48  and  is  extensible  to  process  improvement.  There  are  five  elements  required: 

1.  FAR  clause  52.248-1 

2.  separately  identifiable  systems/software  engineering  work  packages  in  an  earned  value 
system 

3.  baseline  of  prices  for  the  systems/software  engineering  effort 

4.  SCAMPI  (to  establish  process  baseline  and  validate  process  improvements) 

5.  a  SOW  requirement  (to  develop  a  process  improvement  plan  and  to  support  the  periodic 
implementation  of  SCAMPI) 

What  are  the  advantages  of  value  engineering  and  award  fees?  Exercising  the  value 
engineering  clause  could  have  a  greater  financial  reward  potential  than  an  award  fee.  In 
addition: 

•  An  award  fee  requires  an  increase  in  obligation  authority;  value  engineering  does  not. 

•  Value  engineering  requires  visibility  into  systems/software  engineering  work  packages 
and  pricing;  award  fee  application  of  SCAMPI  does  not. 

Ultimately,  an  organization  exercising  the  value  engineering  clause  has  the  potential  to 
demonstrate  that  the  systems/software  engineering  process  improvement  instantiated  the 
resulting  cost  savings  as  well  as  value  added  to  the  products  produced  for  the  sponsoring 
organization. 

The  bottom  line  in  the  brief  discussions  of  award  fee  and  value  engineering  is  that  both 
incentive  approaches  help  management  (sponsoring  organization  and  contractor)  to  focus  on 
overall  process  improvement. 
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3  SCAMPI  Activities  that  Support  Source 
Selection 


3.1  Using  SCAMPI  to  Establish  Baseline  Performance 

The  following  tables  provide  the  essentials  of  SCAMPI  VI.  1  and  corresponding  source 
selection  activities. 

Much  of  the  material  in  these  tables  duplicates  information  from  the  SCAMPI  Method 
Description  Document.  This  information  is  included  in  this  document  for  completeness  of 
this  technical  note  (i.e.,  to  give  the  reader  a  brief,  global  view  of  SCAMPI  relative  to  typical 
source  selection  activities). 


1.1  Analyze  Requirements 


Purpose 


Understand  the  business 
needs  of  the  organization 
for  which  the  appraisal  is 
being  requested.  The 
appraisal  team  leader  will 
collect  information  and 
help  the  appraisal  sponsor 
match  appraisal  objectives 
with  their  business 
objectives. 


Inputs 


Sponsor  requirements 

Initial  requirements  and 

constraints: 

-  appraisal  objectives 

-appraisal  usage  mode 
(internal  process 
improvement,  supplier 
selection,  contract 
process  monitoring) 

-  schedule  and  budget 

-CMMI  reference  model 
representation  and 
domains 

-  organizational  units 
subject  to  appraisal 

-process-related  legacy 
information 


Outputs 


Appraisal  input  consisting 
of: 

-  appraisal  goals  and 
purpose 

-  CMMI  scope 
-organizational  scope 
-appraisal  constraints 

-  sponsor  commitment 


Source-Selection- 
Specific  Activities 


Determine  requirements. 

Initiate  acquisition 
planning. 

Decide  to  use  SCAMPI, 
VI. 1. 


Table  1:  The  Analyze  Requirements  Activity 
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1.2  Develop  Appraisal  Plan 

Purpose  Inputs  Outputs  Source-Selection- 

Specific  Activities 


Document  requirements, 
agreements,  estimates, 
risks,  method  tailoring,  and 
practice  considerations 
(e.g.,  schedules,  logistics, 
and  contextual  information 
about  the  organization) 
associated  with  the 
appraisal.  Obtain,  record, 
and  make  visible  the 
sponsor’s  approval  of  the 
appraisal  plan. 


Appraisal  input  consisting 
of: 

-  appraisal  goals  and 
purpose 

-CMMI  scope 
„  -  organizational  scope 

-  appraisal  constraints 
-sponsor  commitment 


Approved  appraisal  plan 
involving  tailoring  the 
SCAMPI  method, 
identifying  needed 
resources,  determining  cost 
and  schedule,  documenting 
risks  and  resulting  in  a: 

-  strategy  for  managing 
logistics 

-strategy  for  preparing 
organization(s) 

-  schedule 

-  interview  plan 

-  team  assignments 


Seek  sources  using 
Commerce  Business 
Daily  (CBD). 

Develop  the  SSP. 

Document  how  the  source 
selection  will  be 
accomplished. 

Write  the  evaluation  plan 
(EP). 

Develop  the  request  for 
proposal  (RFP). 

The  RFP  requests  and 
delineates  SCAMPI- 
specific  information. 


Definitize  the  SCAMPI 
role  in  source  selection 
(e.g.,  specific  criterion, 
general  consideration).* 

Input  SCAMPI  language 
into  the  RFP.* 

*  These  implementation  activities  generally  occur  in  conjunction  with  SCAMPI  Vl.l  activities  1.2  through  1.4. 


Table  2:  The  Develop  Appraisal  Plan  Activity. 
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1.3  Select  and  Prepare  Team 

Purpose  Inputs 


Ensure  that  an 
experienced,  trained,  and 
appropriately  qualified 
team  is  available  and 
prepared  to  execute  the 
appraisal  process. 


Appraisal  requirements 
and  constraints 

Appraisal  plan 

CMM  scope 

Team  training  materials 


Source-Selection- 
Specific  Activities 


Training  records 

Team  leader  selections 

Team  member  assignments 
and  qualifications 

Prepared  team  that  has 
completed: 

-  method  training 
-model  training 

-  team-building  activates 

-team  orientation 
regarding  appraisal 


Select,  train,  and  prepare 
SCAMPI  personnel  with 
acquisition  requirements 
in  context. 


Table  3:  The  Select  and  Prepare  Team  Activity 


1.4  Obtain  and  Analyze  Initial  Objective  Evidence 
Purpose  Inputs  Outputs 


Obtain  information  that 
facilitates  site-specific 
preparation  and  an 
understanding  of  the 
implementation  of  model 
practices  across  the 
organizational  unit. 
Identify  potential  issues, 
gaps,  or  risks  to  aid  in 
refining  the  plan. 
Strengthen  the 
understanding  of  the 
organization’s  operations 
and  processes. 


Source-Selection- 
Specific  Activities 


Practice  implementation 
data  for  the  organizational 


Identified  participants 

Participants  that  are 
briefed  and  oriented  on 
appraisal  activities 


Completed  instruments  Receive  the  proposal. 


Data  analysis  results  (data 
summaries,  questionnaire 
results,  etc.) 

Identification  of  additional 
information  needed 

Prepared  participants 

Initial  set  of  objective 
evidence 


Evaluate  the  proposals 
initiated. 

Determine  the 
competitive  range. 

Analyze  offerors’ 
SCAMPI  information  for 
establishing  “general” 
prioritization  of  reference 
model  components  for  all 
offerors  relative  to 
objectives  of  the 
acquisition.* 

Finalize  logistical 
coordination  for  site 
visits.* 


*  These  SCAMPI  activities  will  having  timing  and  scheduling  components  tied  directly  to  the  overall  acquisition  schedule. 


Table  4:  The  Obtain  and  Analyze  Initial  Objective  Evidence  Activity 
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1.5  Prepare  for  Collection  of  Objective  Evidence 

Purpose  Inputs  Outputs 


Plan  and  document 
specific  data  collection 
strategies  including: 
sources  of  data,  tools  and 
technologies  to  be  used, 
and  contingencies  to 
manage  the  risk  of 
insufficient  data. 


Appraisal  plan 

Process  implementation 
indicators  (PIIs)  for  the 
organizational  unit 

Initial  objective  evidence 
review 

Data  collection  status 


Confirmation  that 
objective  evidence 
collected  is  sufficient  to 
proceed 

Initial  data  collection  plan 
Replan  of  data  collection 


Source-Selection- 
Specific  Activities 


Depending  on  the  timing 
of  SCAMPI  activities  and 
the  source  selection 
schedule,  you  may  need 
to  support  mid-term 
evaluation  with  initial 
SCAMPI  data  analysis. 


Table  5:  The  Prepare  for  Collection  of  Objective  Evidence  Activity 


2.1  Examine  Objective  Evidence 


Purpose 


Collect  information  about 
the  practices  implemented 
in  the  organization  and 
relate  the  resultant  data  to 
the  reference  model. 
Perform  the  activity  in 
accordance  with  the  data 
collection  plan.  Take 
corrective  actions  and 
revise  the  data  collection 
plan  as  needed. 


Inputs 


Appraisal  data: 

-  initial  objective  evidence 
documents 

-documented  practice 
implementation  gaps 

-  feedback  from 
preliminary  findings 

Data  collection  plan: 

-  appraisal  schedule 

-  interview  schedule 

-  document  list 

-  new  interview  questions 


Outputs 


Updated  appraisal  data 

An  updated  data  collection 
plan 


Source-Selection- 
Specific  Activities 


Continue  to  evaluate  the 
proposals. 

Initiate  the  SCAMPI 
onsite  for  each  offeror. 


Table  6:  The  Examine  Objective  Evidence  Activity 


CMU/SEI-2002-TN-008 


13 


I 


2.2  Verify  and  Validate  Objective  Evidence 


Purpose 


Verify  the 

implementation  of  the 
organization’s  practices 
for  each  instantiation,  and 
validate  the  preliminary 
findings  describing  gaps 
in  the  implementation  of 
model  practices.  Each 
implementation  of  each 
practice  is  verified  so  that 
it  may  be  compared  to 
CMMI  practices,  and  the 
team  characterizes  the 
extent  to  which  the 
practices  in  the  model  are 
implemented.  Gaps  in 
practice  implementation 
are  captured  and  validated 
with  members  of  the 
organization.  Exemplary 
implementations  of  model 
practices  may  be 
highlighted  as  strengths  to 
be  included  in  appraisal 
outputs. 


Inputs 


-schedule  and  participants 
for  data  validation 
activities 

Data  on  practice 

implementation: 

-  strength  and  weakness 
statements 

Data  collection  plan: 

-specifying  additional 
information  needed 


Outputs 


-  notes 

-  strength/weakness 
statements 

-  annotated  worksheets 
Updated  appraisal  artifacts: 

-preliminary  findings 

-revised  data  collection 
plan 

Requests  for  additional  data 


Source-Selection- 
Specific  Activities 


Continue  to  evaluate 
proposals. 

Continue  the  SCAMPI 
onsite  for  each  offeror. 

Preliminary  findings 
presentations  or  focus 
group  interviews  are 
optional,  but 
recommended,  practices 
in  a  SCAMPI 
evaluation.* 


Appraisal  plan:  Updated  appraisal  data: 


*  The  decision  regarding  how  these  practices  will  or  will  not  be  executed  is  made  during  Activity  1.2,  Develop 
Appraisal  Plan. 


Table  7:  The  Verify  and  Validate  Objective  Evidence  Activity. 
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2.3  Document  Objective  Evidence 

Purpose  Inputs  Outputs  Source-Selection- 

Specific  Activities 


Create  lasting  records  of 
the  information  gathered, 
by  identifying  then 
consolidating  notes  and 
transforming  the  data  into 
records  that  document 
practice  implementation 
as  well  as  strengths  and 
weaknesses. 


Appraisal  data: 

-  notes  taken  during  data 
collection  activities 

-  annotated  worksheets  or 
other  work  aids 
containing  data 

-  strengths  and  weaknesses 
documented  from 
previous  activities 

-  data  collection  plan 


Updated  appraisal  data: 

-  noted  practice 
implementation  gaps 

-revised  data  collection 
plan 

-  annotated  worksheets 

Requests  for  additional 
data  (interviewees  or 
documents) 


Continue  to  evaluate 
proposals. 

Continue  the  SCAMPI 
onsite  for  each  offeror. 


Table  8:  The  Document  Objective  Evidence  Activity. 


2.4  Generate  Appraisal  Results 

Purpose  Inputs  Outputs  Source-Selection- 

Specific  Activities 


Rate  goal  satisfaction 
based  upon  the  extent  of 
practice  implementation 
throughout  the 
organizational  unit.  The 
extent  of  practice 
implementation  is 
determined/judged  based 
on  validated  data  (e.g., 
direct,  indirect,  and 
affirmation  objective 
evidence)  collected  from 
the  entire  representative 
sample  of  the 
organizational  unit.  The 
rating  of  capability  levels 
and/or  maturity  levels  is 
driven  by  the  goal 
satisfaction  ratings. 


Appraisal  data:  Final  findings 

-  validated  preliminary  Recorded  rating  decisions 

findings 

-  tabulations  of  objective 
evidence  of  practice 
implementation 

-  annotated  worksheets, 
checklists,  and  working 
notes 


Continue  to  evaluate 
proposals. 

Continue  the  SCAMPI 
onsite  for  each  offeror. 


Table  9:  The  Generate  Appraisal  Results  Activity 
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3.1  Deliver  Appraisal  Results 


Purpose 

inputs 

Outputs 

Source-Selection- 
Specific  Activities 

Provide  credible  appraisal 
results  that  can  be  used  to 
guide  actions.  Represent 
the  strengths  and 
weaknesses  of  the 
processes  in  use  at  the 
time.  Provide  ratings  (if 
planned  for)  that 
accurately  reflect  the 
capability  level/maturity 

Appraisal  data; 

-final  findings 
-ratings 

Appraisal  artifacts: 

-appraisal  input 
-appraisal  plan 

Documented  final  findings 

Final  report  (if  requested) 

Recommendations  report 
(if  requested) 

Continue  to  evaluate 
proposals. 

Continue  the  SCAMPI 
onsite  for  each  offeror. 
(The  source  selection 
process  and  constraints 
may  prevent  the  delivery 
of  appraisal  results  on¬ 
site.) 

level  of  the  processes  in 

use  Incorporate  the  appraisal 

results  into  the  source 
selection  evaluation 
context  to  be  presented  to 
the  SSAC  and  SSA. 

The  SSEB  compares  data 
collected  against  the 
evaluation  standard  and 
assigns  technical  ratings 
and  risk  identifications. 

The  SSAC  compares  and 
ranks  offeror  proposals 
submits  a  risk  assessment 
to  the  SSA. 

The  SSA  makes  the 
award  decision. 


The  SCAMPI  team  may  or  may  not  be  formally  part  of  the  SSEB.  If  they  are  not,  then  the  team  provides  SCAMPI 
findings/outcomes  to  the  SSEB.  The  SCAMPI  team  consults  with  the  SSEB  if  requested.  The  SCAMPI  team  may  act  as 
advisors  to  the  SSAC  and  SSA. 

Table  1 0:  The  Deliver  Appraisal  Results  Activity 
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3.2  Package  and  Archive  Appraisal  Results 


Purpose 

Inputs 

Outputs 

Source-Selection- 
Specific  Activities 

Preserve  important  data 
and  records  from  the 
appraisal,  and  dispose  of 
sensitive  materials  in  an 

Appraisal  data: 

-  appraisal  input 

-  appraisal  plan 

Appraisal  record 

Completed  forms  and 
checklists 

The  appraisal  record  will 
be  tailored  and  sanitized  as 
appropriate  and  agreed  to 
during  planning. 

appropriate  manner. 

-final  findings 

Sanitized  data  (as 

-  objective  evidence 

appropriate  and  agreed 

Appraisal  team  artifacts: 

upon  during  planning) 

-  notes 

Lessons  learned  (appraisal 

-documented  practice 
implementation  gaps 
-  preliminary  findings 
-document  library 

team,  organization) 

Table  1 1:  The  Package  and  Archive  Appraisal  Results  Activity 


3.2  Benefits  of  SCAMPI  in  Source  Selection  and  Contract  Process 
Monitoring 

The  primary  reason  for  using  SCAMPI  in  source  selection  is  to  reduce  the  risk  of  selecting  an 
organization  that  has  immature  and  ineffective  process  and  product  life  cycles.  This  risk 
reduction  activity  enhances  the  achievement  of  a  successful  system/product  development  and 
delivery  to  the  customer.  Using  SCAMPI  in  contract  process  monitoring  enables  the 
customer  to  have  detailed  insight  and  incentive  tools  available  to  ensure  that  a  development 
organization  maintains  its  mature  processes  or  is  making  steady,  measured  progress  in 
achieving  defined  improvement  objectives  relative  to  the  CMMI  model.  While  using 
SCAMPI  to  benchmark  an  organization’s  process  and  product  life  cycles  does  not  necessarily 
guarantee  a  successful  product,  the  likelihood  of  success  should  increase  as  the  processes 
mature.  In  other  words,  mature  processes  reduce  the  risk  associated  with  the  planned  product 
development.  Reduced  risk  is  the  benefit. 


3.3  Schedule  and  Resource  Issues  Related  to  Using  SCAMPI  in 
Source  Selection 

One  of  the  major  issues  related  to  implementing  SCAMPI  in  source  selection  is  the 
compatibility  of  the  source  selection  schedule  with  the  SCAMPI  appraisal  schedule.  The 
typical  SCAMPI  evaluation  takes  approximately  10  working  days  for  the  site  visit  alone 
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(offeror  preparation  work  and  potential  pre-onsite  visit  preparation  requirements  exacerbate 
the  schedule). 

An  acquisition  reform  push  over  the  last  several  years  has  significantly  reduced  source 
selection  schedules  (90  days,  120  days  etc).  Given  the  lead  times  for  source  selection  briefing 
generation  and  coordination  prior  to  presenting  to  the  SSA,  even  less  time  is  available  for  the 
actual  evaluation.  For  example,  consider  the  situation  where  a  planned  acquisition  has  two 
bidders  and  no  subcontractors.  Using  SCAMPI  for  source  selection  is  probably  feasible  in 
nominal  schedule  timeframes.  However,  some  larger  programs  can  have  five  offerors,  each 
having  three  major  subcontractors.  That  equates  to  thirty  weeks  of  site  visit  time.  The  use  of 
SCAMPI  and  scheduling  site  visits  becomes  complex  to  meet  typical  SSA  timeline 
requirements. 

A  balance  must  be  established  between  the  source  selection  schedule  and  the  coverage  to  be 
provided  in  the  SCAMPI  evaluation.  For  example,  the  following  are  ways  that  could  be  used 
to  accommodate  the  SCAMPI  evaluation  in  an  otherwise  short  source  selection  schedule: 

•  The  SCAMPI  evaluation  could  be  tailored  to  look  at  only  a  subset  of  the  process  areas 
using  the  continuous  representation. 

•  Multiple  SCAMPI  teams  could  operate  in  parallel. 

•  Evaluating  subcontractors  could  be  excluded  from  the  SCAMPI  evaluation  (with  the 
attendant  increase  in  risk). 

•  The  source  selection  schedule  could  be  established  to  accommodate  the  level  of  SCAMPI 
thoroughness  required. 

•  The  SCAMPI  team  could  be  excluded  from  the  SSEB  membership.  That  allows 
evaluations  to  start  prior  to  opening  discussions.  This  accommodation,  of  course, 
decreases  the  level  of  insight  and  participation/input  that  the  SCAMPI  team  has  to  the 
rest  of  the  evaluation. 

•  The  full  process  area  coverage  desired  could  be  kept,  but  a  class  B  or  class  C  appraisal 
could  be  performed. 

There  are  probably  other  innovative  approaches  to  address  schedule  imbalance,  and  each 
program  has  its  own  particular  issues  to  address  and  accommodate.  It  is  important,  though,  to 
recognize  early  that  there  may  be  a  schedule  issue  that  affects  your  program  and  you  may 
need  to  devise  a  SCAMPI  approach  that  balances  the  benefits  of  a  shortened  source  selection 
timeline  and  SCAMPI  risk  reduction. 
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4  Next  Steps 


This  is  the  first  attempt  to  explain,  at  a  high  level  of  abstraction,  how  SCAMPI  V  1.1  can  be 
integrated  with  the  typical  acquisition  activities  of  supplier  selection  and  contract  process 
monitoring. 

Each  acquisition  has  its  own  unique  characteristics  and  constraints.  Forethought  and  careful 
planning  judiciously  applied  will  provide  useful  data  from  the  SCAMPI  Vl.l  benchmarking 
appraisal  methodology.  Implementation  guidance  specific  to  the  supplier  selection  and 
contract  process  monitoring  environments  is  expected  to  be  published  as  a  different  version 
or  addendum  to  the  existing  SCAMPI  Method  Definition  Document  or  as  a  separate  Method 
Implementation  Guide. 
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